Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Infer] Add head_dim=96 dispatch for block attention #70763

Merged
merged 1 commit into from
Jan 14, 2025
Merged

[Infer] Add head_dim=96 dispatch for block attention #70763

merged 1 commit into from
Jan 14, 2025

Conversation

zeroRains
Copy link
Contributor

@zeroRains zeroRains commented Jan 10, 2025 •
edited
Loading

PR Category

Inference

PR Types

Bug fixes

Description

为block attention添加head_dim=96的dispatch

复现方式：PaddleNLP中使用下面的指令

cd llm
# block attention
python ./predict/predictor.py --model_name_or_path bigscience/bloom-1b1 --dtype float16 --mode dynamic --decode_strategy greedy_search --inference_model 1 --block_attn 1

# normal attention
python ./predict/predictor.py --model_name_or_path bigscience/bloom-1b1 --dtype float16 --mode dynamic --decode_strategy greedy_search --inference_model 1 --block_attn 0

# block 的指令与normal指令的计算结果相同

@paddle-bot Paddle-bot
Copy link

paddle-bot bot commented Jan 10, 2025

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-bot paddle-bot bot added the contributor External developers label Jan 10, 2025
@zeroRains zeroRains requested a review from SigureMo as a code owner January 11, 2025 09:05
@yuanlehome yuanlehome merged commit 825e6ec into PaddlePaddle:develop Jan 14, 2025
33 checks passed
@zeroRains zeroRains mentioned this pull request Jan 14, 2025
Open
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yuanlehome yuanlehome yuanlehome approved these changes

@SigureMo SigureMo Awaiting requested review from SigureMo

Assignees
No one assigned
Labels
contributor External developers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zeroRains @yuanlehome